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1 .  Executive  Summary 

The  Air  Force  CALS  Test  Bed  (AFCTB)  testing  philosophy  has  been 
to  incorporate  as  many  different  tests  (as  practical  in  the 
testing  process)  to  establish  maximum  confidence  in  the  inter¬ 
pretation  and  implementation  of  CALS  Standards.  Over  time,  this 
has  resulted  in  complex  testing  procedures  with  many  different 
steps.  These  steps  consist  of  software  applications,  operating 
systems,  and  commands;  requiring  the  tester  to  master  them  in 
order  to  complete  the  testing  and  reporting  cycle.  The  purpose 
of  creating  the  automation  procedures  in  the  test  bed  was  to 
shift  the  burden  of  manual  command  entry  and  file  management  by 
the  tester  to  the  automation  software.  This  reduces  the  time, 
effort,  and  expertise  required  to  perform  much  of  the  testing  and 
reporting  tasks.  The  result  is  a  networked,  multiuser  environ¬ 
ment  functional  across  the  different  system  platforms  found  in 
the  AFCTB. 
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2,  Introduction 

The  AFCTB  testing  philosophy  has  been  to  incorporate  as  many 
different  tests  as  practical  in  the  testing  process  to  establish 
maximum  confidence  in  the  interpretation  and  implementation  of 
CALS  Standards.  Over  time,  this  has  resulted  in  complex  testing 
procedures  with  many  different  steps  involving  software  applica¬ 
tions,  operating  systems,  and  commands  required  to  complete  the 
testing  and  reporting. 

This  report  documents  an  effort  to  generalize,  standardize, 
automate,  and  document  these  testing  procedures  for  use  by  users 
less  familiar  with  the  process.  The  tools  resulting  from  this 
effort  are  effective  ways  to  increase  availability,  throughput, 
and  the  ease  of  test  performance  by  prompting  the  users  along  the 
entire  process. 

The  purpose  of  this  report  is  to  document  the  recent  effort  in 
analysis,  systems  integration,  and  automation  of  the  test  proce¬ 
dures  used  in  the  AFCTB  to  evaluate  test  file  adherence  to  CALS 
Standards.  Another  purpose  is  to  document  the  testing  and  re¬ 
porting  activities  performed  by  the  AFCTB  and  other  possible 
facilities  operating  within  similar  hardware/software/networking 
environments.  The  goal  of  this  effort  was  to  increase  the  test¬ 
ing  resource  availability,  ease  of  use,  speed,  and  test/report 
throughput  by  the  following: 

Enabling  resource  sharing  to  the  maximum  extent  possible, 

-  reducing  difficulty  and  errors  by  automating  and  elimina¬ 
ting  typing  of  long,  complex,  and  repetitive  commands, 
data  file  names  and  paths  to  data, 

reducing  time,  effort,  and  expertise  required  to  perform 
a  great  deal  of  the  testing  and  reporting  tasks 
(excluding  expert  interpretation  of  difficult  test  re¬ 
sults  and  modification  of  test  data  for  re -test  --  al¬ 
though  the  automation  does  not  preclude  these  actions,  it 
does  not  propose  to  have  knowledge  or  give  advice  beyond 
the  errors  reported  by  the  testing  software 
applications) , 

integrating  the  entire  test  and  reporting  procedure  from 
start  to  finish  and  guiding  users  along  who  are  unfamil¬ 
iar  with  all  the  CALS  Standards,  hardware  platforms, 
operating  systems,  command  languages,  and  application 
software  used  for  testing  and  reporting,  and 
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performing  integrated  file  management  service  for  the 
users  from  start  to  finish  in  a  networked,  multi-user 
environment,  across  the  different  system  platforms  and 
invoking  the  different  networking  protocols  transparent 
to  users. 


This  tool  automates  the  test  procedures  to  run  TAPETOOL,  AUTOCAD, 
SGML  parsers  and  the  other  tools  by  executing  a  set  of  C- Shell 
scripts  written  in  the  UNIX  C- Shell  command  language  and  a  set  of 
batch  files  written  in  the  MS-DOS  batch  command  language.  The 
user  starts  a  top  level  procedure  by  typing  its  name.  This 
procedure  and  all  related  sub-procedures  invoke  the  appropriate 
applications,  tape  procedures,  networking  protcols,  remote  file 
copying,  file  transfer,  placement  of  the  files  in  the  user's 
account,  and  file  cleanup. 

3.  CALS  Standards  Testing  Applications  Automated/Integrated 

The  following  are  the  CALS  Standards  tests  and  applications 
currently  automated,  integrated,  and  available  per  workstation: 


CALS  Standard 

Applications 

Resident 

/ 

Availabilitv 

MIL-STD-1840A 

TAPETOOL 

Sun  3/280 

All 

MIL-D-28000 

(IGES) 

Rosetta  Preview 

Sun  3/60 

Sun  3/60 

IDA  Parser 

Cheetah 

486 

PC 

/Comp.  PCs 

IDA  Verify 

Cheetah 

486 

PC 

/Comp .  PCs 

CADKEY/IG2C 

Cheetah 

486 

PC 

/Comp .  PCs 

AUTOCAD 

Cheetah 

486 

PC 

/Comp .  PCs 

MIL-M-28001 

(SGML) 

Exoterica  XGML 

Cheetah 

486 

PC 

/Comp .  PCs 

MIL-R-28002 

(Raster) 

VALIDG4 

Sun 

VALIDG4 

Cheetah 

486 

PC 

/Comp .  PCs 

Hijaak 

HJ2  Viewer 

Cheetah 

486 

PC 

/Comp .  PCs 

Cheetah 

486 

PC 

/Comp .  PCs 

MIL-D-28003 

(CGM) 

Metacheck/Metacals 

Cheetah 

486 

PC 

/Comp.  PCs 

Harvard  Graphics 

Cheetah 

486 

PC 

/Comp .  PCs 

Novell  Server 

/Comp .  PCs 

QSTR  Test  Reports 

Enable 

Novell  Server 

/Comp .  PCs 

Cheetah 

486 

PC 

/Comp .  PCs 
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This  table  identifies  the  tools  that  are  currently  being  utilized 
in  the  AFCTB.  If  your  environment  uses  some  or  all  of  these 
tools,  they  will  be  documented  in  the  automated  procedures.  If 
your  environment  has  no  need  for  a  tool,  simply  delete  those 
steps  from  the  appropriate  procedure.  Likewise,  to  add  a  new 
tool  to  the  procedure,  model  existing  steps  with  the  appropriate 
names. 

The  networking  protocols  used  were  both  Telnet  (TCP/IP)  to  the 
Suns  and  Novell  IPX/SPX  providing  LAN  Client/Server  service  to 
PCs  running  on  top  of  a  10  Mbps  Ethernet.  If  this  setup  differs 
from  yours,  simply  replace  our  network  protocols  and  commands 
involving  file  transfer  with  the  appropriate  commands  that  fit 
your  configuration. 

Use  of  the  above  suite  of  platforms/software  tools  and  tests 
greatly  helps  in  establishing  confidence  in  the  interpretation 
and  implementation  of  the  CALS  Standards  whenever  the  same  re¬ 
sults  are  achieved  in  these  different  environments.  Conversely, 
different  results  help  flag  possible  errors  in  interpretation  of 
the  data  and  help  provide  feedback  to  software  vendors. 


4.  Analysis  of  Previously  Existing  Test  Environment 

Analysis  of  the  previously  existing  test  environment  resulted  in 
the  following  findings: 

-  Access  was  limited  only  to  those  who  were  very  fluent  in 
both  DOS  and  UNIX  command  language  and  knew  how_  and  where 
to  go  to  execute  commands  and  how  and  when  to  invoke  the 
different  networking  protocol  commands  (to  transfer  files 
for  exair^le)  . 

Users  must  be  very  familiar  with  the  order  of  running 
test  procedure  application  programs  and  be  able  to  navi¬ 
gate  from  one  directory  to  another,  copying  appropriate 
files  as  they  go.  Meanwhile,  two  or  more  users  could  not 
copy  files  for  an  application  without  overwriting  each 
other's  files.  They  must  then  clean  up  those  files  when 
finished.  The  LAN  File  Server  was  not  being  fully  uti¬ 
lized  to  access  data  files  in  individual  user  accounts 
and  support  a  multi-user  environment. 

Users  must  type  in  most  commands  by  hand  and  know  the 
names  and  locations  of  both  data  files  and  executable 
files  including  CALS  testing  applications  and  other 
commands  such  as  batch  files,  with  increased  chance  of 
errors . 
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5.  Enhancements  Enabled  Through  Integration/Automation  Effort 

To  Start  with,  this  automation  software  is  considered  to  be  a 
Beta  Test  version  and  not  the  final  product,  utilizing  only  the 
software  products  currently  available  on  the  platforms  and  not 
yet  incorporating  other  requested  utility  software.  The  automa¬ 
tion  software  is  not  an  "end  to  all  ends"  in  automating  CALS 
testing  and  reporting,  i.e.  it  DOES  NOT  do  the  following: 

invoke  analysis  and  reporting  beyond  the  extent  that  the 
CALS  applications  generate  analysis  and  reports. 

expert  analysis  and  reporting,  i.e.  interpretation  of 
graphical  or  text  output  generated  by  the  applications 
from  the  data  files. 

automatic  correction  of  errors  and  re- test.  However,  it 
does  enable  an  expert  user  to  modify  test  data  and  retest 
to  see  the  effect. 

currently  generate  complete  test  reports  (QSTRs) .  This 
is  a  planned  enhancement.  In  any  case,  the  final  docu¬ 
ment  may  still  require  expert  review  and  editing.  The 
QSTR  document  would  then  also  be  automatically  tagged  per 
its  DTD  and  put  into  the  appropriate  database. 

It  DOES  however  achieve  the  following  goals: 

Enables  increased  resource  sharing. 

Enables  multi-user  environment. 

Increases  ease  of  use,  speed,  and  throughput  of  test  data 
through  CALS  suites  of  test. 

Eliminates  the  need  to  be  ]aiowledgeable  of  CALS  Stan¬ 
dards,  test  procedures,  application  software  (to  some 
extent),  operating  systems,  and  networking  protocols, 
from  a  user  who  wishes  to  run  tests.  Therefore,  a  large 
portion  of  the  test  cases  are  made  available  to,  and  can 
be  accoitplished  by,  a  much  larger  set  of  possible  users. 

Integrates  the  entire  test  and  reporting  procedure  from 
start  to  finish  and  guides  the  user  along. 

Performs  integrated  file  management  service  for  all 
users . 
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6.  Conclusions 

The  systems  integration  and  automation  effort  and  software  have 
produced  an  efficient  way  to  handle  increasing  flows  of  CALS  test 
data  by  enabling  more  availability  and  potential  throughput.  It 
is  believed  that  continued  integration  and  automation  efforts 
such  as  this  are  the  tools  which  will  yield  higher  productivity 
and  growth  in  these  iir5)ortant  activities,  yet  keeping  resource 
requirements  under  control . 

These  tools  are  currently  available  on  3.5"  or  5.25"  floppy  disk 
media  upon  request  by  contacting  the  Air  Force  CALS  Test  Bed  at 
(513)  257-3085  or  DSN  787-3085. 
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